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METHOD AND SYSTEM FOR INDEXING Real-time note taking systems work as long as the visual 

AND CONTROLLING THE PLAYBACK OF representation is formed by a sequence of element-creating 

MULTIMEDIA DOCUMENTS events. However, problems arise when the representation is 

edited during the course of a meeting. For example, a user 

This is a Division of application Ser. No. 08/959,527 5 may delete and/or change events. However, a deletion 

filed Oct. 28, 1997 now U.S. Pat. No. 5,986,655. The entire removes the element and the associated time stamp. When a 

disclosure of (he prior application is hereby incorporated by new element is substituted for the deleted element, the time 

reference herein in its entirety. stamp of the new element reflects the time that the new 

BACKGROUND OF THE INVENTION e J ement ^ added > u rather than ^ e time °{ or *F? al 

element. Therefore, the corresponding time of the underly- 

1. Field of Invention m g event is lost. This occurs because the indices are the 
This invention relates to electronic multimedia document visible elements and not the underlying events in the meet- 
systems. In particular, this invention is a method and a ing. 

system for providing an historical index to a multimedia Another disadvantage of the conventional real-time note 

document using temporal icons. More specifically, this taking system is that a person must be devoted to the 

invention provides temporal icons that present temporal 15 note-taking task. It is desirable to create an index into the 

information about corresponding events or timestreams and meeting recording without dedicating a person to the task of 

also provides complete playback control through a gestural note taking. The indexing should be a byproduct of the 

interface on the temporal icons. meeting or activity. 

2. Description of the Related Art ^ Historical data is captured by multimedia document sys- 
The collection of audio, video, and other records of a tems using one or more capture devices, such as an audio 

meeting provides a rich, reviewable record of the history of recorder or an electronic whiteboard. These recordings typi- 

a meeting or a presentation. Unfortunately, the benefits of cally have timestreams and events. Timestreams are sets of 

such a record is often outweighed by the unwieldiness of time-stamped continuous data streams which represent some 

sequential access. The advent of digital recordings has 25 recordable aspect of a meeting. Typical timestreams include, 

provided the ability to instantly access any portion of a for example, audio and video signals, and/or a history of the 

recording. However, instant access is only valuable to the inputs on an electronic whiteboard. Events are occurrences 

user if the user knows which portion of the recording to within a timestream. For example, an event may be a 

access and how to gain access those portions. particularly significant action taken on the electronic 

A user of a multimedia document system will typically 30 whiteboard, such as an input of typed or handwritten notes, 

want to playback the recording of a meeting to recall page changes, highlighted items, etc., or may be a change in 

conversations and/or other activities that occurred during the a speaker during the session. Events can be used to create 

meeting. Such a user may include one of those who were indices which provide direct access to a point in time during 

present during the meeting or may be one who did not attend a meeting. Timestreams may inherently define events, or 

the meeting. 35 alternatively, may be analyzed to identify events. Event 

One conventional approach to indexing a multimedia information includes a time stamp, an event type and a list 

document is termed real-time note taking. Real-time note of various properties of the instance of the event. Indices 

taking describes a technique whereby a person takes notes tne timestream can be created by a post-session analysis 

during a meeting using a system that time stamps the various such as °y» for example, a speaker identification system 

individual elements of the notes. A real-time note taking 40 analysis on an audio stream. 

system provides a comprehensible index into the recording Documents recorded by multimedia document systems, 

of the meeting. A user can retrieve and play a part of the include, handwriting on an electronic whiteboard, typed or 

recording simply by selecting a corresponding element of handwritten notes, annotated materials, etc. These docu- 

the notes. ments may have been produced and recorded during a 

The simplest example of a real-time note taking system is 45 meeting. However, the documents may not have been pre- 

a text transcription system. The user of a text transcription sented during the meeting but may have been generated by 

system simply types the text as the meeting proceeds. The a post-session analysis and recorded as an integral part of the 

text can be a literal transcription of what is being said or can meeting recording. 

be arbitrary notes about the meeting. The notes include a When reviewing the recording, conventional multimedia 

string of text, and the time stamped note elements are the 50 document systems provide both the documents and the 

individual characters of the text. The notes can also be indices to a user. Typically, a user will read the documents 

"strokes", "ink", "digital ink" or "marks" input by a pen and selectively replay some of the recorded materials. The 

based input device. Each stroke of the pen on the display user may read the documents to understand the content of 

generates a corresponding digital ink mark. The digital ink the meeting and refer to the indices to understand the 

marks are time-stamped and each mark serves as an index 55 chronological sequence of events of the meeting, 

element. After the note taking is complete, the user can A problem for user interface design in multimedia docu- 

select a character or element and the corresponding part of ment systems is the presentation of the indices to a user, 

the meeting record will be retrieved and played. Conventionally, there are two different approaches. The first 

Conventional note taking systems allow users to construct approach represents the events on a timeline. For example, 

a visual representation of indices into the meeting recording. 60 some systems use colored segments of a timeline to show the 

The visible representation in this case are the notes. Each speaker during different points in time. However, the user of 

index element has a time associated with it, which is the time such a system is required to manage and coordinate at least 

when the element was created. Since each index element is two different representations. The first representation is the 

visible, the user selects a character by simply pointing to it. timeline of the events and the second is the display of the 

Because the character has an associated time, the user 65 documents. 

obtains access to the corresponding point in the meeting The second approach to user interface representation of 

recording. indices implicitly represents the indices of the documents in 
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a manner similar to the real-time note taking systems. For BRIEF DESCRIPTION OF THE DRAWINGS 

example, each stroke drawn on an electronic whiteboard „ t t t . „ 

may be time-stamped during the meeting. The time- J The preferred embodiments of this invention will be 

stamping makes each stroke an index into the recording. described in detail, with reference to the following figures, 

However, significant strokes cannot be distinguished from 5 wnerein * 

insignificant strokes. Therefore, a user is forced to adopt FIG- 1 is a block diagram of one embodiment of the 

other methods or systems to keep track of the significant system of this invention; 

strokes, such as by using, for example, bullets or stars. FIG. 2 is a block diagram of one embodiment of the 

However, these indices cannot be edited and the history of processor of FIG. 7. 

the recording is not displayed to the user. 10 FIG. 3 shows an example of a presentation slide with 

Furthermore, multiple times are often associated with a analog clock icons as the temporal icons in accordance with 

document element, such as the time an element was created, one embodiment of the method and system of this invention; 

and the time it was last edited. There is no conventional FIG 4 shows tne elation states Q f the temporal icons in 

method or system for associating multiple times with index accordance with one embodiment of the method and system 

elements. is c f this invention; 

One multimedia document system uses a clock icon as a FIG. 5 shows the play states of the temporal icons in 

temporal icon and index into a recording. Each clock icon accordance with one embodiment the method and system of 

indicates a single point in time. Tapping on a clock icon ^ mve ntion- 

causes the document to start play back at the time indicated z L ' lL , . , r 

on the clock icon. However, These clock icons only indicate » . .' 6 shoWS the P' 0 *^. States ^f™ 1 

a single point in time. In particular, they do not indicate ^ons m accordance with one embodiment of the method and 

spans of time. Therefore, a user cannot determine how long s y stem of mis lDventl0n i 

a specific topic was addressed during a meeting. A user of FIGS - 7A ~ 7D 1S a flowchart outlining the operation of the 

this multimedia document system only knows at what point control routine of one embodiment of the method and 

in time the discussion of an issue may have been initiated. 25 system of this invention. 

Additionally, these clock icons do not indicate the por- DETAILED DESCRIPTION OF PREFERRED 

tions of the recording which have been reviewed and those EMBODIMENTS 
portions which have not been reviewed. Therefore, a user 

might replay portions of a document that have already been This invention provides an index for the replay of records 

reviewed when the user is intending to review only previ- 30 of collaborative activities, such as meetings, and other 

ously unreviewed portions of the recording. The conven- non-collaborative events such as a presentation. While the 

tional clock icon indicates the point in time where a discus- invention is described in the setting of a meeting, the method 

sion of an issue was initiated. However, it does not indicate and system of this invention is not limited to such activities, 

the progress of the review through the span of time in which Non-collaborative activities include presentations, speeches, 

the issue was discussed. Therefore, a user is forced to 35 training sessions and the like. 

speculate on the progress of a review through a portion of Meetings among groups of people are held for various 

the document. reasons, such as, for example, brainstorming, project review, 

The conventional clock icons also do not provide any etc. Often, a participant, or some other designated person, 

control over playback, other than the ability to initiate the ^ will take minutes of the meeting to report the results of the 

playback at the point in time associated with the clock icon. meeting. The creation of such minutes typically require that 

A user is required to access a separate graphical user written notes be maintained. However, such written notes of 

interface (hereafter GUI) that resembles a conventional tape the meeting may be incomplete, and taking the written notes 

deck or VCR control panel. may interfere with the meeting. Other meeting artifacts, such 

SUMMARY OF THE INVENTION 45 as > for example diagrams or lists, are also created through 

mLU,Ttl111 ^ use of some shared representation medium, such as an 

This invention provides a system and a method that use electronic whiteboard. In any event, reviewing such, notes 

temporal icons to index and control playback of a multime- and/or artifacts is often desirable when preparing the meet- 

dia document. This invention further provides temporal ing minutes. 

icons that indicate the point in time of an event and the time 5Q Another aid in creating meeting minutes is an audio or 

span of a timestream. This invention also provides temporal video recording. Unfortunately, these recordings are purely 

icons that indicate the state of the playback of the associated sequential. Thus, it is often very difficult to efficiently 

events and timestreams. retrieve information from such recordings. 

This invention additionally provides a method and a The system and method of this invention create indices 

system that enable gestural control of playback functions. 55 into a meeting recording of a collaborative activity using the 

One embodiment of the method and system of this natural byproducts of the activity. The system and method of 

invention uses analog clock icons as the temporal icons. An this invention provide an intuitive way for people to gain 

analog clock icon indicates an indexed point of time using direct access to portions of the recorded session. Events such 

its hands and a span of time using an arc on its bezel. The as a change of speaker, writing on, or merely putting 

analog clock icons also indicate their states using colors and 60 markings on, an electronic whiteboard, and taking a note are 

shades of the face of the analog clock icons. The analog examples of such natural byproducts of a meeting that 

clock icons can be created automatically or manually using provide useful index points. 

controls on the display. Gestures made on the clock icons FIG. 1 is a block diagram of one embodiment of the 

provide complete playback control of the document. electronic multimedia document system 10 of this invention. 

These and other features and advantages of this invention 65 The electronic multimedia document system 10 has a pro- 
are described in or are apparent from the following detailed cessor 12 communicating with an input/output interface 14. 
description of the preferred embodiments. The input/output interface 14 communicates with any num- 
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ber of conventional input/output devices, such as a slide electronic whiteboard creates a timestream that is a history 

projector 16, a pointer 18, a loudspeaker 20, a mouse 22, a file of time stamped primitive operations, 

microphone 24, a pen 26, a keyboard 28, and an electronic An important aspect in operating an electronic whiteboard 

whiteboard 30. The input/output devices 16-30 provide an are "gestures". In general, most interactions on the elec- 

interface to input and output data and to input control signals 5 t r0 nic whiteboard are "ink strokes" that create marks or 

from a user of the electronic multimedia document system "ink" displays on the electronic whiteboard. Thus, "ink 

10. strokes" are analogous to marking a regular whiteboard with 

The processor 12 also communicates with any number of an ink marker. A "gesture" is an interaction with the elec- 
local storage devices such as a hard drive 32 and a memory tronic whiteboard that is created like any other "ink stroke", 
device 34. The processor 12 may also communicate with an 10 However, a "gesture" differs from a regular "ink stroke" in 
external data source 36 across a communication line 38 that that it is interpreted as a command to the electronic white- 
may have an interruptible connection 40. board rather than a mark on the electronic whiteboard. For 

FIG. 2 is a block diagram showing one embodiment of the example, if a particular "object" formed by one or more "ink 

processor 12 in greater detail. The processor 12 is preferably strokes" is to be deleted from the display on the electronic 

implemented using a general purpose computer 100. The 15 whiteboard, the "object" is first selected by an encircling 

general purpose computer 100 preferably includes an editing gesture with the pen. The encircling gesture is then finished 

system 102, a playback system 104, a playback controller witn a "delete" gesture, such as a "pigtail". After being 

106, an instantiation system 108, a controller 110 and a "deleted", the object encircled by the "delete" gesture no 

memory 112. The elements of the general purpose computer lon S er appears on the display of the electronic whiteboard. 

100 are interconnected by a bus 114. The editing system 102, 20 When analyzing a timestream captured by an electronic 

the playback system 104, the playback controller 106 and whiteboard, an "event" of the timestream drawing a figure, 

the instantiation system 108 controlled by the controller 110, writing a word, can include changing a page, erasing figures 

are used to implement the flowchart shown in FIGS. 7A-7D, or words, making notations relative to existing marks, or 

described below. It should be appreciated that the editing entering a particular command to the electronic whiteboard 

system 102, the playback system 104, the playback control- 25 interface. 

ler 106 and the instantiation system 108 are preferably FIG. 3 shows an example of a presentation slide 59 having 

implemented as software routines running on the controller clock icons 50 that indicate when each point 52 on the 

110 and stored in the memory 112. It should also be presentation slide 59 was discussed. The system and method 

appreciated that many other implementations of these ele- of this invention create indices using temporal icons such as 

ments will be apparent to those skilled in the art. 30 the clock icons 50. The temporal icons are embedded within 

As shown in FIG. 2, the system 10 is preferably imple- the context of the multimedia document being indexed. The 

mented using a programmed general purpose computer. temporal icons are used throughout a multimedia document 

However, the system 10 can also be implemented using a to show how the multimedia document proceeds through 

special purpose computer, a programmed microprocessor or 35 time. 

microcontroller and any necessary peripheral integrated Each temporal icon indicates both a point in time and a 

circuit elements, an ASIC or other integrated circuit, a span of time. Therefore, the temporal icons reveal to the user 

hardwired electronic or logic circuit such as a discrete how the document was created and used over a period of 

element circuit, a programmable logic device such as a PLD, time. The user can access this information merely by glanc- 

PLA, FPGA or PAL, or the like. In general, any device on 4Q ing at the temporal icons in the multimedia document, 

which a finite state machine capable of implementing the Each temporal icon acquires its attributes from its asso- 

flowchart shown in FIGS. 7A-7D can be used to implement ciation with corresponding elements of the multimedia 

the processor 12. document. This association can be represented by the spatial 

Additionally, as shown in FIGS. 1 and 2, the memory relationships of the temporal icons to the other elements on 

device 34 and 112 are preferably implemented using static or 45 a display of the multimedia document. Thus, the significance 

dynamic RAM. However, the memories 34 and 112, and the of the attributes of a temporal icon positioned near a note on 

hard disk 32, can also be implemented using a floppy disk the display is explained by the contents of the note. As 

and disk drive, a writable optical disk and disk drive, a hard shown in FIG. 3, the temporal icon 54 positioned near the 

drive, flash memory or the like. Additionally, it should be point 52 labeled "ENGINEERING" 56 in the presentation 

appreciated that the memories 32, 34 and 112, in various 50 slide 59 indicates when and how long the point 52 labeled 

combinations, can be either distinct portions of a single "ENGINEERING" 56 was discussed. The temporal icons 58 

memory or physically distinct memories. positioned at the top of the slide indicate when and how long 

Furthermore, it should be appreciated that the communi- the slide 59 of the presentation was displayed during the 

cation line 38 can be a wired or wireless link to a network. meeting. More than one temporal icon 58 at the top of the 

The network 36 can be a local area network, a wide area 55 slide 59 indicates that the slide 59 was displayed more than 

network, the Internet or any other distributed processing and once during a meeting. 

storage network. The temporal icons 50 and 58 can be created by a user by 

The input/output devices 16-30 preferably capture the gesturing on an electronic whiteboard or by hitting a special 

timestream in a digital format. The timestream is desirably key on a keyboard. The temporal icons 50 and 58 can also 

stored in a digital format to facilitate direct and random 60 be created automatically by the system 10. For example, the 

access to the various parts of the timestream. However, system 10 can put a new temporal icon 58 at the top of the 

capturing and storing the timestream data in an analog slide 59 every time that slide 59 is displayed during a 

format is also within the scope of this invention. In general, meeting. 

the format of the timestream need only correspond to the The temporal icons 50 and 58 can be edited to redefine 

input/output device that was used to create the timestream. 65 their time attributes, so that they become more accurate 

For example, an audio timestream is formed by a set of audio indices. For example, additional temporal icons 50 and 58 

samples taken at some regular interval. In contrast, an can be added to a document during replay to add additional 
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indices to the document. In this case, a temporal icon 50 or 
58 acquires its time attributes from the playback time of the 
recorded session, rather than the time that the session was 
recorded. For example, during a meeting, if an issue is raised 
and discussed, a user of the electronic whiteboard can create 
an index to the discussion of that issue by creating a clock 
icon 50 and writing a note. However, in this example, the 
clock is usually created after the discussion of the issue 
began. Therefore, the clock can be edited to adjust its time 
backwards to more accurately index the start of the discus- 
sion. This editing can take place during the meeting, or can 
occur during a later access session. 

During an access session, the meeting is conveniently 
displayed from the current clock setting, to determine if the 
setting is accurate. For the user's convenience, the replay is 
starts automatically after an edit is performed. Therefore, the 
user gets immediate feedback on the appropriateness of the 
time setting. 

One embodiment of the clock icon represents and displays 
several time attributes for each document element. The clock 
icon indicates an element's time span, including its begin- 
ning and ending times, its creation state, its play state, and 
its play progress state. 

One embodiment of the clock icon 50 indicates the 
starting time using analog clock hands. The clock icon 50 
indicates the time span using an arc 60 positioned along the 
clock face's outer bezel 62. The clock icon indicates its state 
using various colors, shades and stripes. 

As shown in FIG. 3, in one embodiment, a span of time 
is represented on the clock icon 50 using arcs 60 on the 
bezels 62. Each arc 60 begins at the minute hand of the clock 
and sweeps out the length of time of the corresponding 
timestream. The span can be edited by double tapping on the 
clock icon 50 during recording or playback. A double tap 
displays a graphical user interface that enables editing of the 



time attributes, by tapping or gesturing on the primordial 
clock using the remote pointer during a meeting or a 
presentation. 

The clock icons 50 and 58 can take any of several states, 
5 as shown in FIGS. 4-6. These states include creation states, 
play states, and play progress states. 

The creation states, shown in FIG. 4, indicate the stage of 
creating a clock icon. In the "normal" creation state 70, a 
clock icon has a time span arc 76 indicating a beginning time 
10 72 and an ending time 74. The "live" creation state clock 
icon 78 initially has only a beginning time. The beginning 
time is indicated by the clock hands and initially does not 
have an ending time. The ending time is eventually set by the 
user or automatically by some event, such as a slide change. 
A live clock icon 78 has its face highlighted and does not 
have an arc on its bezel. A clock icon in the primordial 
creation state 80 has no time attributes associated with it, 
until it is instantiated during a meeting. A primordial clock 
icon 80 thus has a bezel 82 but does not have hands or time 
spans. Therefore, a primordial clock icon 80 generally 
resembles a bullet. 

FIG. 5 shows several "play" states for the temporal icons 
50 and 58. The play states indicate whether a clock icon is 
playable or not, and whether it is currently being played. The 
face of a clock icon 84 that is being played has a first color, 
25 the face of a clock icon 86 that is not being played, but is 
playable, has a second color. The face of clock icon 88 that 
is not playable has a third color. The clock icon 88 may not 
be playable because, for example, it has a time span that is 
out of the current range of the recording. 

FIG. 6 shows several "play progress" states for the 
temporal icons 50 and 58. The play-progress state indicates 
which portions of a clock icon's time span has been played 
during a particular session. The clock face 90 has a fourth 
color to represent the portions of the timestream correspond- 



20 



30 



time attributes of the clock icon 50. The user also does not ^ m f to the of the clo L ck «»n M that have been played, 

have to specify an end point for each clock icon 50. Rather, 0f course > * ere are L other methods to display the play 

when the user creates a new clock icon 50, the end time of Pj°gress. such as a thermometer-like icon just below each 

the previous clock icon 50 is set. clock. 

Gestures on the temporal icons 50 can also be used to Playing straight through the time span fills the face of the 

control playing back a multimedia document. The temporal 40 clock icon 90 from the bottom up. The play-progress state 



icons provide full access and control of the temporal ele- 
ments of the meeting. No other devices, such as a VCR-type 
control panel or timeline, are needed. The user can focus 
directly on the temporal icons to control access to both the 
content and the temporal process. A document with temporal 45 
icons can also easily be combined with a timeline in a 
document. 

The clock icons 50 and 58 can be created both by the user 
and by the system. As explained above, a clock icon 58 is 
generated by the system automatically each time the dis- 
played slide 59 changes. Each clock icon 58 indicates both 
when and how long the slide 59 was viewed. These clock 
icons 58 are placed in a row at the top of the slide, as shown 
in FIG. 3. 

A user creates a clock icon 50 or 58 using a "circle" 



50 



also contains the clock icon's current play point, which is 
marked by a horizontal line 96. Playing portions of the clock 
icon by skipping back and forth to other clock icons fills the 
clock face 92 with stripes 94. The relative vertical position 
of a stripe 94 indicates its relative temporal location in the 
total span. The bottom of the clock icon represents the 
beginning of the span of time and the top of the clock icon 
represents the end. In contrast to a timeline which has a 
single current play point, a document with a collection of 
clocks has multiple current play points. 

Preferably, the first color is red, the second color is white, 
the third color is gray, and the fourth color is green, although 
any colors can be used. It should be understood that, while 
the above description described specific colors as indicating 
various clock icon states, the method and the system of this 



gesture. The location of the "circle" gesture on the electronic 55 i ? v 1 ^ n . tion _ is not io} * limited to tn fs e colore. One of ordinary 



whiteboard, for example, indicates the placement of the 
clock icon. The current time (i.e., the time of creation) is 
shown by using the clock hands. When the clock icon 50 or 
58 is created during a playback mode, the clock icon 50 or 
58 indicates the current playback time. 

A remote device, such as a pen on a remotely-operated 
electronic whiteboard or a laser pointer, can be used to create 
the clock icons 50 and 58 during a presentation. To make 
creation easier and placement of the clock icons more 



skill in the art understands that there are many methods of 
indicating the state of the temporal icons of this invention 
and that any method or system is appropriate as long as the 
states are distinguishable from each other. 
60 To control playback using the clock icons 50 and 58, the 
user gestures directly on the clock icon 50 or 58. A tap starts 
the playback from the clock icon's current play point, or 
instantiates a primordial clock icon. If a clock icon is 
currently playing, then a tap pauses the playback. If a clock 



precise, "primordial" clocks, i.e., clocks without time 65 icon is live, then a tap sets the end time. A double tap sets 
attributes, can be created in advance, during preparation of the ending point of a clock icon's time span to be the current 
a document. These "primordial" clocks can be provided with time or the play time. A gesture to the left moves the clock 
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icon's current play point back ten seconds. A gesture to the 
right moves the clock icon's current play point ahead ten 
seconds. An "upward" gesture moves the play point to the 
previous clock icon boundary. A "hold" gesture causes a 
graphical user interface (not shown) to be displayed, to 5 
enable editing the clock icon's time attributes. 

FIGS. 7A-7D outline one preferred embodiment of the 
control routine of the method and system of this invention. 
The control routine starts at step S100 and continues to step 
S110. At step S110, the control routine determines if an 10 
"open document" command has been received. If, at step 
S110, the "open document" command has been received, 
control continues to step S120. Otherwise, control loops 
back to step S110. 

At step S120, the control routine opens a document. Next, 15 
at step S130, the control routine determines if a "record" 
command has been received. If, at step S130, a "record" 
command has been received, control continues to step S140. 
Otherwise, control jumps to step S320, shown in FIG. 7C. At 
step S140, the control routine begins recording the events 20 
and timestreams of the collaborative session, meeting or 
presentation. 

Then step S150, the control routine determines if an index 
point of a primordial clock icon has been encountered such 
as an event or timestream. If, at step S150, an index point of 25 
a primordial clock icon is encountered, control continues to 
step S160. Otherwise, control jumps directly to step S180. 
At step S160, the control routine determines if an event or 
timestream corresponding to the primordial clock icon has 
been encountered. If, at step S160, the control routine 
determines that an event or timestream corresponding to the 30 
primordial clock icon has been encountered, control contin- 
ues to step S170. Otherwise, control again jumps directly to 
step S180. At step S170, the control routine instantiates the 
primordial clock icon by assigning the time attributes of the 
corresponding event or timestream to the clock icon. Control 35 
then continues to step S180. 

At step S180, the control routine determines if a "circle" 
gesture has been made on the display of the clock icon. If, 
at step SI 80, a "circle" gesture is not made, control contin- 
ues to step S190. Otherwise, control jumps to step S240, as 40 
shown in FIG. 7B. 

At step S190 the control routine determines if a "stop 
recording" command has been received. If, at step SI 90, a 
"stop recording" command has been received, control con- 
tinues to step S200. Otherwise, control jumps back to step 45 
S150. At step S200 the control routine stops recording the 
multimedia document. Next, at step S210, the control rou- 
tine determines if a "close document" command has been 
received. If, at step S210, a "close document" command has 
been received, control continues to step S220. Otherwise, 5Q 
control jumps back to step S130. At step S220, the control 
routine closes the current document. Then in step S230, the 
control routine stops. 

As shown in FIG. 7B, at step S240, the control routine 
generates and displays a clock icon. Then, at step S250, the 
control routine determines if a "tap" gesture has been made 55 
on the display of a clock icon. If, at step S250, the control 
routine determines that a "tap" gesture has been made on the 
display of the clock icon, control continues to step S260. 
Otherwise control returns to step S150 of FIG. 7A. 

At step S260, the control routine sets the begin time of the 60 
clock icon to the current time. Next, at step S270, the control 
routine determines if a second tap gesture has been made on 
the display of a clock icon. If, at step S270, the control 
routine determines that a "tap" gesture has not been made on 
a display of a clock icon, control continues to step S280. 65 
Otherwise, control jumps to step S310. At step S280 the 
control routine determines if a "circle" gesture has been 
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made on the display of the clock icon. If, at step S280, the 
control routine determines that a "circle" gesture has been 
made, control continues to step S290. Otherwise, control 
jumps back to step S290. 

At step S290, the control routine sets the end time of the 
current clock icon to the current recording time. Next, at step 
S300, the control routine generates a new clock icon on the 
display at the position of the "circle" gesture. Control then 
returns to step S250. 

At step S310, the control routine sets the end time of the 
clock icon to the current recording time. Control then returns 
to step S150 of FIG. 7A. As set forth above, if, at step S130 
of FIG. 7A, the control routine determines that a "record" 
command has not been made, control jumps to step S320, 
shown in FIG. 7C. At step S320, the control routine deter- 
mines if a "circle" gesture has been made on the display of 
the clock icon. If, at step S320, the control routine deter- 
mines that a circle gesture has been made on the display of 
the clock icon, control continues to step S330. Otherwise, 
control jumps directly to step S340. At step S330, the control 
routine generates a clock icon at the position of the circle 
gesture. Control then continues to step S340. 

At step S340, the control routine determines if a "tap" 
gesture has been made on the display of the clock icon. If, 
at step S340, the control routine determines that a "tap" 
gesture has been made on the display of the clock icon, then 
the control routine continues to step S350. Otherwise, con- 
trol jumps to step S480, as shown in FIG. 7D. 

At step S350, the control routine begins playing back the 
open document. Next, at step S360, the control routine 
determines if a "tap" gesture has been made, on the display 
of the clock icon. If, at step S360, the control routine 
determines that a "tap" gesture has been made, control 
continues to step S370. Otherwise, control jumps to step 
S380. 

At step S370, the control routine stops playing back the 
current document. Control then returns to step S320. 

At step S380, the control routine determines if a "left" 
gesture has been made on the display of the clock icon. If, 
at step S490, the control routine determines that the "left" 
gesture has been made, control continues to step S390. 
Otherwise, control jumps directly to step S400. At step 
S390, the control routine moves back the current play point 
ten seconds. The control routine then continues to step S406. 

At step S400, the control routine determines if a "right" 
gesture has been made on the display of the clock icon. If, 
at step S400, the control routine determines the "right" 
gesture has been made, control continues to step S410. 
Otherwise, control jumps directly to step S420. At step 
S410, the control routine sets the current play point forward 
ten seconds. The control routine then continues to step S420. 

At step S420, the control routine determines if an "up" 
gesture has been made on the display of the clock icon. If, 
at step S420, the control routine determines that the "up" 
gesture has been made on the display of the clock icon, 
control continues to step S430. Otherwise, control jumps 
directly to step S440. At step S430, the control routine 
moves the current play point to the previous clock icon 
boundary. The control routine then continues to step S440. 

At step S440, the control routine determines if a "down" 
gesture has been made on the display of the clock icon. If, 
at step S440, the control routine determines that the "down" 
gesture has been made, control continues to step S450. 
Otherwise, control jumps directly to step S460. At step S450 
the control routine moves the current playpoint to the next 
clock icon boundary. The control routine then continues to 
step S460. 

At step S460, the control routine determines if a "double 
tap" gesture has been made on the display of the clock icon. 
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If, at step S460, the control routine determines that a "double 
tap" gesture has been made, control continues to step S470. 
Otherwise, control jumps directly back to step S360. At step 
S470, the control routine sets the end time of the current 
clock icon to the current play time. The control routine then 5 
also returns to step S360. 

As shown in FIG. 7D, if at step S340, the control routine 
determines that a "tap" gesture has not been made, control 
jumps to step S480. At step S480, the control routine 
determines if a "double tap" gesture has been made on the 10 
display of the clock icon. If, at step S350, a double tap has 
been made, control continues to step S490. Otherwise, 
control jumps directly to step S500. At step S490, the control 
routine sets the end time of the clock icon to the current time. 
Control then continues to step S500. 

At step S500, the control routine determines if a "left" 
gesture has been made on the display of the clock icon. If, 
at step S500, the control routine determines that a "left" 
gesture has been made, control continues to step S510. 
Otherwise, control jumps directly to step S520. At step 
S510, the control routine moves the play point of the current 20 
clock icon back ten seconds. Control then continues to step 
S520. 

At step S520, the control routine determines if a "right" 
gesture has been made on the display of the clock icon. If, 
at step S520, the control routine determines that a "right" 25 
gesture has been made, control continues to step S530. 
Otherwise, control jumps directly to step S540. At step 
S530, the control routine moves the play point of the current 
clock icon forward ten seconds. The control routine then 
continues to step S410. 30 

At step S540, the control routine determines if an "up" 
gesture has been made on the display of the clock icon. If, 
at step S410, the control routine determines that an "up" 
gesture has been made, control continues to step S550. 
Otherwise, control jumps directly to step S560. At step 35 
S550, the control routine moves the current play point to a 
previous clock icon. The control routine then continues to 
step S430. 

At step S560, the control routine determines if a "down" 
gesture has been made on the display of the clock icon. If, 40 
at step S430, the control routine determines that a "down" 
gesture has been made, control continues to step S570. 
Otherwise, control jumps directly to step S580. At step 
S570, the control routine moves the play point to the next 
clock icon. The control routine then continues to step S580. 45 

At step S580, the control routine determines if a "hold" 
gesture has been made, on the display of the clock icon. If, 
at step S450, a "hold" gesture has been made, control 
continues to step S590. Otherwise, control jumps directly 
back to step S210. At step S590, the control routine displays 5Q 
a pop-up graphical user interface that allows the user to edit 
the time attributes of the clock icon. The control routine then 
also returns to step S210. 

While this invention has been described in conjunction 
with the specific embodiments outlined above, it is evident 
that many alternatives, modifications, and variations are 55 
apparent to those skilled in the art. Accordingly, the pre- 
ferred embodiments of the invention as set forth above are 
intended to be illustrative and not limiting. Various changes 
may be made without departing from the spirit and scope of 
the invention as defined in the following claims. 60 

What is claimed is: 

1. A method for indexing a multimedia document for 
subsequent presentation, wherein the multimedia document 
is to expected to have at least one of an event and 
timestream, the method comprising. 65 

generating a primordial temporal icon that is associated 
with the expected at least one of the event and the 



timestream, the primordial temporal icon having at 
least one undefined temporal attribute; and 
embedding the primordial temporal icon in the multime- 
dia document at a time prior to a beginning of the 
timestream, such that, in response to the expected at 
least one of an event and timestream occurring with 
respect to the multimedia document, at least one of the 
at least one undefined temporal attribute will be 
defined; 

wherein the at least one of the event and the timestream 
is independent of, and occurs after and independently 
of, a creation time of the primordial temporal icon. 

2. The method of claim 1, further comprising: 
providing at least one of the event and the timestream that 

corresponds to the expected at least one event and 
timestream; and 
associating at least one time attribute corresponding to the 
at least one of the event and the timestream to the 
primordial temporal icon. 

3. The method of claim 2, wherein associating at least one 
time attribute comprises associating a time of an event. 

4. The method of claim 2, wherein associating at least one 
time attribute comprises associating a beginning time of a 
timestream. 

5. The method of claim 2, wherein associating at least one 
time attribute comprises associating an ending time of a 
timestream. 

6. The method of claim 2, wherein associating at least one 
time attribute comprises associating a length of time of a 
timestream. 

7. The method of claim 1, wherein the primordial tem- 
poral icon comprises a representation of a clock without 
time attributes. 

8. A system for indexing a multimedia document, the 
multimedia document expected to have at least one of an 
event and a timestream, the system comprising: 

a primordial icon generator that generates a primordial 
temporal icon corresponding to the expected at least 
one of the event and the timestream, and that embeds 
the primordial temporal icon in the multimedia docu- 
ment at a time prior to a beginning of the timestream, 
the primordial temporal icon having at least one unde- 
fined temporal attribute; and 

a display that displays the primordial temporal icon and a 
representation of the expected at least one of the event 
and the timestream, such that, in response to the 
expected at least one of an event and timestream 
occurring with respect to the multimedia document, at 
least one of the at least one undefined temporal attribute 
will be defined; 

wherein the at least one of the event and the timestream 
is independent of, and occurs after and independently 
of, a creation time of the primordial temporal icon. 

9. The system of claim 8, further comprising: 

a presentation system that presents the multimedia docu- 
ment and that establishes at least one of an event and a 
timestream corresponding to the expected at least one 
of the event and the timestream; and 

an instantiation system that associates at least one time 
attribute of the at least one of the event and the 
timestream to the primordial temporal icon. 

10. The system of claim 8, wherein the primordial tem- 
poral icon comprises a representation of a clock without 
time attributes. 



08/03/2004, EAST Version: 1.4.1 



